The most frequent 1000 words contain a many stopwords and the most frequent content words. Assuming that stopwords are usually short, the list presented here shows some important content words.
This list can be ordered not only by word length (as in this subsection), but also by rank or alphabetically. This will be done in the next subsections.
The content words in the table give a first impression of the main subject areas in a corpus.
In the case of very poor pre-processing, some non-words may appear in the list. But there will be much finer tests for poor pre-processing below.
The list above can be generated from a relatively small corpus. If we have a small corpus which is representative for a larger one, the corresponding results should not differ very much. Hence, the list could be generated quickly generated from a small randomly chosen subcorpus.
SELECT @num:=@num+1 as local_rank, w_id-100 as rank_in_wordlist,word,char_length(word) as len from (select @num:=0) xx, words where w_id>100 and w_id<=1100 order by len desc limit 50;
Longest Words in Top-1000 by rank
Longest Words in Top-1000 alphabetically